In recent years, research into so-called mixed reality (MR) systems, representing a seamless conjunction between real space and virtual space, has been actively pursued.
In general, there are two main types of MR systems: A video see-through system, in which an image in virtual space (such as a virtual object or text information drawn by computer graphics, for example (hereinafter CG)) is superimposed on an image of real space that has been sensed by an image-sensing apparatus such as a video camera or the like, and an optical see-through system, in which real space is optically transmitted through the display screen of a display apparatus and the virtual space image is overlappingly displayed on that display screen.
There is a wide array of applications of this type of MR, which is qualitatively completely different from hitherto existing virtual reality systems. Such applications include navigation systems that overlappingly display names and directories on buildings and the like on real streets, as well as landscape simulations that overlappingly display a computer graphic image of the building scheduled to be erected at a building construction site.
What these various applications have in common is the need for precise, accurate alignment between the real space and the virtual space, and conventionally a variety of efforts have been made to satisfy this requirement.
The problem of alignment in video see-through systems MR boils down to the problem of obtaining a 3-dimensional position/attitude of the perspective of the image-sensing apparatus within a global coordinate system set to real space (hereinafter simply called “global coordinate system”). The problem of alignment in optical see-through systems MR can also be said to similarly boil down to the problem of obtaining a 3-dimensional position/attitude of the perspective of the observer (user) within a global coordinate system.
One common method of solving the above-described problem involves the use of position/attitude sensors such as magnetic sensors, ultrasonic sensors or the like to obtain a 3-dimensional position/attitude of the image-sensing apparatus or observer perspective within the global coordinate system.
Similarly, in outdoor settings, under circumstances in which there is sufficient distance between the image-sensing apparatus or observer and the object so that a fixed value may be used for the position of the perspective, the problem of alignment boils down to the problem of obtaining the 3-dimensional attitude of the perspective. In this case, 3-dimensional attitude sensors composed of a combination of gyro sensors and accelerometers are used to obtain the 3-dimensional attitude.
The output values output by the 3-dimensional attitude sensors, is the 3-dimensional attitude of the sensors themselves in the sensor coordinate system. Here, in case of Tokimec Inc. attitude sensors TISS-5-40, for example, a sensor coordinate system is a coordinate system that defines the Y axis as the opposite direction of the pull of gravity and the -Z axis as a frontal direction of a sensor when activated atop an X-Z plane defined by this Y axis. In this manner, the output values output by the 3-dimensional attitude sensors are typically not the 3-dimensional attitude of the perspective in a global coordinate system that is the measurement object. In other words, the sensor output values themselves cannot be used as a 3-dimensional attitude of the perspective in a global coordinate system but must undergo some sort of coordinate conversion. More specifically, the sensor attitudes must be coordinate-converted to the perspective attitudes and attitudes in the sensor coordinate system must be converted to attitudes in the global coordinate system. It should be noted that, in the present specification, the data for carrying out coordinate conversion between the sensor output values and the 3-dimensional attitude of the perspective in a global coordinate system is called correction information.
FIG. 1 shows the structure of a conventional image display apparatus for displaying MR via the optical see-through system.
As shown in the diagram, a display screen 110 is a see-through type, and together with an attitude sensor 120 is fixedly mounted on a head attachment part 100. When a user (not shown in the diagram) attaches the head attachment part 100 so that the display screen 110 is positioned before his or her eyes, the user can observe the real space in front of the display screen through an optical system (not shown in the diagram) of the display screen 110. The attitude sensor 120 measures its own attitude in the sensor coordinate system and outputs attitude measurements in three degrees of freedom. It should be noted that the attitude sensor 120 has an internal inclinometer (not shown in the diagram) capable of measurement in the direction of the pull of the earth's gravity, and as described above, one of the axes of the sensor coordinate system (in the present example, the Y axis) is set so as to be in the reverse direction of the pull of gravity.
The attitude information output unit 130 changes the measurements input from the attitude sensor 120 in accordance with the correction information retained in a memory 140. The attitude information output unit 130 computes the attitude of the user's perspective within a global coordinate system and outputs the result as perspective attitude information. An image generator 150 generates a virtual image corresponding to the user's perspective attitude in accordance with the perspective attitude information input from the attitude information output unit 130 and outputs that virtual image. The display screen 110 then displays the virtual image input from the image generator 150. With such a configuration, the user can see a virtual image displayed on the display screen 110 superimposed on an image of real space via the display screen 110.
Next, using FIG. 2, a description will be given of a method of calculating the user perspective attitude in a global coordinate system with the attitude information output unit 130.
As shown in FIG. 2, the attitude of a sensor coordinate system 210 in a global coordinate system 200 is denoted as RTW, the attitude of the attitude sensor 120 in the sensor coordinate system 210 is denoted as RST, the relative attitude of the user perspective 220 as seen from the attitude sensor 120 is denoted as RVS, and the attitude of the user perspective 220 in the global coordinate system 200 is denoted as RVW.
Here, R is a 4×4 matrix, where RBA describes the attitude of an object B in a given coordinate system A. In other words, this matrix is a coordinate conversion matrix for converting from a coordinate system A to a coordinate system B that an object B defines, and defines the conversion formula PB=RBAPA that converts the coordinates PA=(XA, YA, ZA, 1)T in coordinate system A to coordinates PB=(XB, YB, ZB, 1)t in coordinate system B. That is, the user's perspective 220 in the global coordinate system 200 denoted as RVW can be said to be a coordinate conversion matrix (Pv=RVWPW) for the purpose of converting the global coordinate system 200 coordinates PW=(XW, YW, ZW, 1)T into the user's perspective coordinate system 230 coordinates PV=(XV, YV, ZV, 1)T.
The Matrix R is the product of a rotation matrix Rx defined by an angle of rotation θ about the X axis, a rotation matrix Ry defined by an angle of rotation (angle of direction) φ about the Y axis, and a rotation matrix Rz defined by an angle of rotation ψ about the Z axis, so as to satisfy the equation R=RzRxRy. Each of these matrixes is expressed as follows:
                                          R            ⁢                                                  ⁢                          x              ⁡                              (                θ                )                                              =                      [                                                            1                                                  0                                                  0                                                  0                                                                              0                                                                      cos                    ⁢                                                                                  ⁢                    θ                                                                                        sin                    ⁢                                                                                  ⁢                    θ                                                                    0                                                                              0                                                                                            -                      sin                                        ⁢                                                                                  ⁢                    θ                                                                                        cos                    ⁢                                                                                  ⁢                    θ                                                                    0                                                                              0                                                  0                                                  0                                                  1                                                      ]                          ,                                  ⁢                              R            ⁢                                                  ⁢                          y              ⁡                              (                ϕ                )                                              =                      [                                                                                cos                    ⁢                                                                                  ⁢                    ϕ                                                                    0                                                                                            -                      sin                                        ⁢                                                                                  ⁢                    ϕ                                                                    0                                                                              0                                                  1                                                  0                                                  0                                                                                                  sin                    ⁢                                                                                  ⁢                    ϕ                                                                    0                                                                      cos                    ⁢                                                                                  ⁢                    ϕ                                                                    0                                                                              0                                                  0                                                  0                                                  1                                                      ]                          ,                                  ⁢                              R            ⁢                                                  ⁢                          z              ⁡                              (                ψ                )                                              =                      [                                                                                cos                    ⁢                                                                                  ⁢                    ψ                                                                                        sin                    ⁢                                                                                  ⁢                    ψ                                                                    0                                                  0                                                                                                                        -                      sin                                        ⁢                                                                                  ⁢                    ψ                                                                                        cos                    ⁢                                                                                  ⁢                    ψ                                                                    0                                                  0                                                                              0                                                  0                                                  1                                                  0                                                                              0                                                  0                                                  0                                                  1                                                      ]                                              Formula        ⁢                                  ⁢        1            
where RVW can be shown byRVW=RVS—RST—RTW  (Formula A)
Here, the Y axis of the sensor coordinate system 210 is set in a direction that is the reverse of gravity, so by defining the Y axis of the global coordinate system 200 as perpendicular to the plane of the horizon, the directions of the Y axes of the global coordinate system 200 and the sensor coordinate system can be made to coincide. At this time the components around the X axis and Z axis of RTW, that is, RxTW and RzTW, each become unit matrixes, with RTW becoming equivalent to the rotational matrix RyTW defined by the angle of rotation φTW around the Y axis. Therefore, Formula A described above changes toRVW=RVS—RST—RyTW  (Formula B)
where RST is the input from the attitude sensor 120 to the attitude information output unit 130 and RVW is the output from the attitude information output unit 130 to the image generator 150. RVS and RyTW (in other words, the angles of rotation θVS,φVS and ψVS about the three axes that define RVS and the angle of rotation φTW that defines RyTW) correspond to correction information needed in order to convert RST to RVW. Using the RST input from the attitude sensor 120 as well as the RVS and RyTW retained in the memory 140, the attitude information output unit 130 calculates RVW based on Formula B described above and outputs same to the image generator 150.
In order to align real space and virtual space accurately, correction information must be accurately set in the memory 140 by some means. Only after accurate correction information has been provided can display of a virtual image accurately positioned in real space be achieved.
One commonly known method for setting correction information involves a user or an operator dialogically increasing and decreasing each of the values for θVS, φVS, ψVS and φTW stored in the memory 140 via an input means not shown in the diagram and adjusting the values through a process of trial and error until an accurate alignment is achieved.
However, with such a method, four parameters must be adjusted simultaneously, which is difficult and time-consuming.
One method of reducing this difficulty is a method proposed in Japanese Patent Application No. 2001-050990 (which corresponds to U.S. patent application Ser. No. 09/818,600 filed on Mar. 28, 2001), a description of which is given below.
FIG. 3 is a block diagram of an image display apparatus when this correction information setting method is incorporated into the image display apparatus of FIG. 1.
As shown in the diagram, the present composition is the same as that shown in FIG. 1, though with the addition of a correction information calculator 310, an instruction input unit 320 and a switching unit 330. In the image display apparatus of FIG. 3, the compositional elements that correspond to the attitude information output unit 130 as well as the memory 140 perform differently from their counterpart elements in the image display apparatus of FIG. 1, so these elements are labeled attitude information output unit 130′ and memory 140′ in recognition of that functional difference.
The calculation of the correction information is performed by moving the attitude of the user's perspective 220 to attitude Ry0VW and obtaining the attitude sensor 120 output R0ST at that point. It should be noted that attitude Ry0VW is a particular preset attitude in the global coordinate system, and has a visual axis that is horizontal to the plane of the horizon. The memory 140′ holds the preset perspective attitude Ry0VW (or the rotation φVW about the Y axis defining it)).
The switching unit 330 receives input from the user or operator (neither being shown in FIG. 3) and sets the mode of the attitude information output unit 130′ to either a normal mode or a correction information calculation mode.
When set to the normal mode, the attitude information output unit 130′, as with the attitude information output unit 130 described with reference to FIG. 1, calculates RVW using correction information from the RST input from the attitude sensor 120, and outputs same to the image generator 150 as perspective attitude information.
If the attitude information output unit 130′ is set to the correction information calculation mode, however, then the attitude information output unit 130′ inputs Ry0VW from the memory 140′ and outputs same to the image generator 150 as perspective attitude information. The instruction input unit 320 receives the input from the user or operator and transmits an instruction to execute a correction information calculation to the correction information calculator 310. More specifically, the user or operator, in the correction information calculation mode, adjusts the attitude of the perspective 220 until the virtual image displayed on the display screen 110 and the image of real space observed by optically seeing through the display screen 110 are in the correct positional relationship. Then, when the user or operator determines that both images sufficiently overlap each other (that is, at the point at which the user or operator determines that the perspective 220 is positioned at the attitude Ry0VW), he or she inputs that information to the instruction input unit 320 by, for example, pressing a particular key.
The correction information calculator 310 inputs the instruction to calculate correction information from the instruction input unit 320, inputs output R0ST of the attitude sensor 120 at that point (that is, at the point at which the user or operator determines that the perspective 220 is positioned at the attitude Ry0VW), and calculates the correction information based on the attitudes Ry0VW and R0ST.
With the method described above, the Y axis rotational component RySV (or the angle of rotation φSV around the Y axis that defines the Y axis rotational component RySV) of the reverse matrix RSV (which expresses the relative attitude of the attitude sensor 120 as seen from the user perspective 220) of the attitude RVS that is one piece of correction information must already be given by some method, and already stored in the memory 140′.
At this time, from the data processed by the correction information calculator 310, Formula B provides thatRy0VW=RVS—R0ST—RyTW  (Formula C)
By varying Formula C, we getRy0VW=(RzVSRxSVRySV)−1R0STRyTW  (Formula D)
The correction information calculator 310 inputs Ry0VW and RySV from the memory 140′, inputs R0ST from the attitude sensor 120, and calculates the unknown correction information RzSV, RxSV and RyTW based on Formula D. The following shows the steps in that process.
That is, by further varying Formula D, we getRzSVRxSVRySVRy0VW=Rz0STRx0STRy0STRyTW  (Formula E)
Both the left side and right side of the above equation have been multiplied by the rotational components of the X, Y, and Z axes, so identical equations exist for each of the rotational components of the Z, Y, and X axes.
First, the identical equations for rotational components of the Z and X axes, are obtained, which are as follows:RzSV=Rz0ST  (Formula F)RxSV=Rx0ST  (Formula G)
From the foregoing RzSV and RxSV can be obtained.
Similarly, the identical equation for the rotational component of the Y axis is as follows:RySVRy0VW=Ry0STRyTW
from whichRyTW=RySVRy0VWRy0ST−1  (Formula H)
and from which RyTW can be obtained.
The correction information calculator 310 calculates the correction information RzSV, RxSV, and RyTW using the processes described above, and from these values further calculates RVS (=(RzSVRxSVRySV)−1), and outputs RVS and RyTW (or the angles of rotation θVS, ψVS and φTW that define these) to the memory 140′.
It should be noted that the derivation of RySV may be done through a process of trial and error using measurements made with a protractor or the like, or by using some other means.
As described above, merely by fixing the perspective attitude in three degrees of freedom at a predetermined attitude, unknown correction information can be easily extracted and it becomes possible to achieve accurate alignment.
So long as the display screen 110 and the attitude sensor 120 remain fixed relative to each other, the RVS part (or the angles of rotation θSV and ψSV) of the correction information remains unchanged. Thus, each time correction information is extracted (that is, when RVS becomes known), the only correction information that changes together with each use of the system is RyTW (or the angle of rotation φTW). However, with each use of the system the method described above requires the performance of the same operations as would be needed to extract all the correction information (that is, the operations required to fix the perspective in three degrees of freedom at a predetermined attitude).
However, a problem with the above-described method is that it is not easy to fix the perspective in three degrees of freedom at a predetermined attitude, and any deviation between the predetermined attitude and the actual attitude at the time the correction information is input appears as accidental error in the correction information. For this reason, the above-described correction calculations must be performed repeatedly, until adequate alignment is achieved.